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ABSTRACT 

Counseling center efforts to establish and maintain 
relevant effectiveness in the services they offer, must be based on 
valid and reliable feedback from critical sources. Record keeping and 
data collection are the procedures by which this feedback can be 
acquired if they are designed to answer questions and do not become 
ends in themselves. A follow-up questionnaire and mailing procedure 
designed to provide both descriptive and evaluative data can be most 
useful in producing data useful in a number of important ways in the 
assessment of agency response to environmental needs. Cross-agency 
comparisons have great potential and emphasize the importance of the 
development of a data bank equipped to handle both descriptive and 
evaluative material. The Counseling Services Assessment Blank was 
recently developed and adopted, along with a standardized mailing for 
the evaluation of counseling center feedback through a data bank 
being developed at Colorado State University. This data bank will 
make possible agency comparisons of evaluative data for the first 
time. Feedback assessment of the type described here is essential for 
improvement in the effectiveness of both individual staff and agency 
programs. (Author) 
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Environmental flux and change inevitably plays a critical role in determining 
both direction and effectiveness of Counseling Center services. It» Just as 
inevitably, creates two sources for potential error in center efforts to establish 
and maintain relevant effectiveness. The first type of error is the result of 
the center that builds an acceptable program of services, procedures, and policies, 
and then works to maintain that existing program. This center soon finds itself 
out of phase with environmental needs and demands and eventually becomes an agency 
offering irrelevant services on the one hand, and failing to offer badly needed 
services on the other. The second source of error applies to the center that 
recognizes the need for ongoing adjustments and modifications in response to 
environmental change, but makes them in the absence of accurate feedback so that 



they are in essence haphazard and often incongruent with environmental demands. 

In the first case, the center simply stagnates and is left behind. In the second, 
modification occurs, but not in response to the valid criteria of environmental 
feedback. 



Valid and reliable feedback then, is an essential ingredient in all efforts 
to evaluate what exists, what ought to be implemented, and the effectiveness of 
that implementation. In a counseling center, this evaluation is relevant to the 






growth and effectiveness of both individual staff members and the center program 



jNas a whole. Many centers have at one time or another become concerned with 



lO 



evaluating the effectiveness of: 
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Individual and group counseling in general 
Testing programs and the use of tests in counseling 
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Vocational-Occupational information sources 
Agency image as it is projected on campus 
Description of the clientele served 
Physical facilities 

Needs and demands of the campus environment 

Special service programs (e.g., skills Improvement workshop) 

In addition to such agency services* it is not uncommon for individuals on a staff 
to attempt an ongoing systematic assessment of their own individual effectiveness. 

An agency program evaluation may also be designed so that individual staff members 
have the option of retrieving data specific to the effectiveness of their 
performance . 

Record keeping and data collection are the procedures by which a systematic 
and objective evaluation may occur. A number of surveys reveal that these 
procedures are common to almost all centers in one form or another and generate 
basically two kinds of data. The first is descriptive, in that it delineates 
such things as number of clients seen, nature and frequency of presenting problems, 
average number of sessions per client, client demography, nature and frequency 
of test usage, student to counselor ratio, etc. These data are useful in predicting 
and specifying future counselor load, prevalent patterns of student concern, 
quantitative Impact of the agency on the campus population, future staffing pat- 
terns and demands, etc. This is the kind of data that center directors find 
essential in their efforts to justify Increases in staff, salary, and service 
programs . 

The second kind of data gathered by counseling centers is evaluative, in that 
it is designed to provide information pertinent to assessing the effectiveness of 
the service described by the first kind of data. For example, do those students 
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whose effectiveness Is hindered by test anxlel;y receive counseling that reduces that 
anxiety and thereby improve grades? Do those students reporting loneliness and 
frustration in their Inability to establish satisfying Interpersonal relation- 
ships acquire those skills and attitudes necessary for success? And do those 
students requesting assistance with the selection of a major and future vocational 
development receive help in discovering the nature of their aptitudes and how 
they relate to the world of work? The difficulties that plague any attempt to 
assess the effects of psychotherapy also plague the attempts to collect meaningful 
data of this, and the fact that center directors less often find evaluative data 
useful In justifying the existence or expansion of services, probably explain the 
comparative lack of continuing, systematic attempts to collect It In centers 
across the country. And yet, without feedback of this nature, the process of 
learning to Increase both program and Individual effectiveness Is seriously 
hindered . 

The accumulation of both descriptive and evaluative data may be utilized In 
a number of Important ways. Longitudinal comparisons may be made as the body 
of data builds over time and can be helpful in determining whether the nature of 
the clientele served differs from fall quarter to spring — or from the fall 
quarter of the preceding year. The analyses of data collected over a number of 
years may also provide an assessment of trends In service requests, and the 
center's response (or lack of it) to those trends. The center's role In terms 
of whether its focus Is on situational adjustment as opposed to psychotherapy may 
for example, be assessed through the average number of sessions per client. One 
of the most Important aspects of data collection and utilization lies In coordi- 
nating the descriptive with the evaluative. For example, descriptive data might 
reveal that an increasing number of students are requesting help with academic 
skills Improvement and this could be interpreted as Indicative of a developing 



student need on campus. Evaluative feedback relative to how effectively the 
center is responding to these requests would be essential In the modification or 
development of a skills Improvement program sponsored by the center. Finally, 
data of an evaluative nature are prerequisite to basic research Into the theory 
and practice of counseling. Counseling Centers across the country should recog~ 
nize a professional obligation to sponsor studies of this type. 

Perhaps the greatest potential for data use, however, lies in inter-agency 
comparisons through the development of data banks* The data bank developed by 
Tom Magoon for the American College Personnel Associations Commission on 
Counseling is an example of an effort in that direction. That particular bank, 
however, does not provide for the collection of both descriptive and evaluative 
data. Its main emphasis is only the description of existing programs, policies, 
staffing patterns, etc. The only question relating to evaluation simply asks if 
there is such an effort in effect in the respondents agency. Whereas, the ACPA 
data bank has demonstrated its usefulness for cross-agency comparisons of 
descriptions of programs and services, it is not designed to handle cross- 
agency comparisons of staff and agency effectiveness in the implementation of 
those programs. A data bank dealing with both description and evaluation of 
counseling agencies is being developed at Colorado State University. 

Procedures for the collection of data vary widely according to the resources 
available and the nature of the feedback desired. Evaluation may take the form 
of assessing specific programs and projects, or assessing many or all of the 
agency services offered on a routine basis with a single instrument in one gather- 
ing. Examples of the former include a study of the impact on college students of 
counseling informational brochures (Bigelow, Hendrix, and Jensen, 1966), a 
study designed to provide a follow-up evaluation of group and individual counsel- 
ing (Hewer, 1967), and a study investigating student perceptions of a university 
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counseling center (Mlnge and Cass, 1966). Whereas, such evaluations are 
relatively limited In scope and therefore easier to complete, they are Inefficient 
as an ongoing method In that they are usually not carried out regularly and the 
preparation process must be repeated with each new study undertaken. Omnibus 
studies, on the other hand, are designed to evaluate all or most of the crucial 
and routinely offered services In one coordinated effort. Once an instrument has 
been developed and the data collection process established, th&Be evaluations can 
take place routinely with a minimum of effort and result in the production of a 
variety of data useful In many different ways. Through a number of agencies 
adopting a standardized Instrument and collection technique, and pooling their 
data, meaningful cross-agency comparisons can be made. A systematic procedure 
for counseling center description, evaluation, and comparison has been developed 
at Colorado State University. By adopting a questionnaire designed to generate 
both descriptive and evaluative data, specifying a sampling procedure to be 
followed In the use of the Instrument, and then making provision for the pooling 
of the results, cross-agency assessments of both description and evaluation Is 
possible for the first time. 

The Questionnaire 

The Counseling Services Assessment Blank (CSAB) was recently developed for 
use In omnibus evaluations of services provided by counseling agencies. The 
Instrument was designed for compatibility with the Missouri Diagnostic 
Classification Plan (MDCP) . and elicits from respondents self-diagnoses In terms 
of the MDCP categories. Client's ratings of satisfaction with agency services and 
perceived self-growth are elicited according to the MDCP categories also so that 
they may be assessed in relation to the specific problems they see themselves 
as having. 

The MDCP Is a two-dimensional system of diagnostic constructs for use In 
counseling. It was constructed by Berezin (1957), and refined and tested by 
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Apostal and Miller (1959). Subsequent research by Callls and Clyde (1960) > Myers, 
Johnson, and Cacavas (1960),Klrk (1962), Borresen (1963, 1965), Callls (1965), 
Shepard (1965), and Weigel, Cochenour, and Russell (1967), makes the MDCP one of 
the best researched of all counseling diagnostic plans. It is presented In 
Figure 1. Problem~Goal refers to the content of the problem for which the client 
desires assistance. The dimension. Cause, refers to the underlying causal factors 
of the content problem. The three by five scheme makes possible fifteen different 
diagnostic cells, each of which bears a different two-digit coding. The first 
digit represents the Problem-Goal , the second digit, the Cause dimension. In 
practice, both a primary and secondary diagnosis are typically employed. It 
should be noted that although the CSAB Is compatible with the MDCP , Its usefulness 
Is In no way dependent upon the use of that particular diagnostic system. 

The CSAB provides a demographic description of clientele which Includes name, 
student number, age, sex, marital status, class In school, academic major, and 
the number of sessions the client was seen. The agency services that are evalu- 
ated Includes testing programs, confidentiality, perceptions of overall Individual 
and group counseling experience, perceptions of counselor and group members* 
helpfulness, physical facilities, client-perceptions of the agency's Image and 
role In the institution, and a dialogue response. Hurst, Weigel, Thatcher, 
and Nyman (In press) , demonstrated the utility of the CSAB In using It to provide 
the data necessary for an Investigation of the relationship of counselor-client 
diagnostic agreement to perceive outcomes of counseling. They found the mean 
satisfaction score for each MDCP category uniquely helpful In revealing a specific 
phase of the agency program that was Ineffective in response to an expressed 
student need. In adopting the CSAB for use In the Colorado State University data 
bank, a flexible, productive questionnaire was chosen (flexible and productive 
exchanges will be possible.). 
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bampllng Procedure 



Ctocts-agency comparisons o£ service effectiveness can only be meaningful if 
the procedure for securing the feedback has an optimum degree of standardization. 

This Is also true for Intra-agency longitudinal studies where data collection 
occurs over a period of time. Standardization, however, Is not the only reason 
for concern with the sampling technique. The effectiveness of a beautifully developed 
sampling Instrument can be negated by an Inadequate or misleading mailing technique 
and/or cov^r letter. Calahan and Meier (1939) were among the first to recognize 
the differential effects that particular mailing procedures have on responses to 
questionnaires. Research by Suchman and McCandless (1940) suggests that mall 
survey techniques may also have a significant effect on who ans era the question- 
naires In the first place. One of their most Important conclusions Is that an 
Increase In the number of responses is directly related to a decrease In the bias 
of the resulting data. Robin (1965) surveyed literature pertinent to the possible 
differences between respondents and nonrespondents to mall questionnaires and 
found that on most variables there seems to be no difference, but In investigations 
concerned with opinions, values, and psychological characteristics, apparent 
differences are more likely. This possibility Is of critical Importance for 
counseling agencies utilizing the CSAB with a mailing technique and further 
emphasizes the Importance of an adequate return rate as well as a standardized 
procedure for meaningful cross-agency comparison. 

Method of Mailing 

The general content and appearance of the series of letters used in the 
mailing Is of critical Importance. All letters should be written with an emphasis 
on Informality and couched In terms of a personal appeal for help In the evaluation 
process. Efforts should be made to point out the unique Importance of the 











1 






1 



recipient In having information and opinions critical to the evaluation of the 
agency* The appearance of a personal letter can be attained by typing a master 
copy of the body of the letter, using it to reproduce the necessary number of 
letters for the survey with a photocopying technique, and typing the individual 
address and salutation v/ith the same typewriter and ribbon used to type the 
master. Checking with staff members to assure the appropriateness of first name 
usage in the salutation and a live signature by a figure of legitimate authority 
(e.g. agency director) adds to the effectiveness of the letters. The importance 
of a personal plea in increasing response rate in survey t;jechnlques has been well 
demonstrated by Robinson and Agislm (1951), Longworth (1953), and others. The 
content of the follow-up letters should be written to shift emphasis from the 
evaluation itself to the importance of the client responding the questionnaire. 

The system for securing an adequate response rate to mail questionnaires 
adopted for the Colorado State University data bank is described below and is 
an adaptation of Robin's survey (1965) of a combination of methods. Essentially, 
the procedure consists of a minimum of two and a maximum of six contacts with 
potential respondents. 

1. The first contact may be a pre-questionnaire letter sent prior to the I 

questionnaire itself. The contents explain that the respondent will be receiving 

an evaluation questionnaire shortly and that he is in a unique position to make 
a contribution to the evaluation. It also contains a simplified explanation of 
the evaluation rationale, its importance, and possible application. The assurance { 

of confidential handling of all information is important. 

2. The second contact, sent after no more than seven days, may consist of 
the questionnaire and a cover letter. The cover letter may remind the potential 
respondents of the previous letter, repeat its contents, and emphasizes the 
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Inclusion of a stamped, self-addressed envelope. The respondent is thanked for 
his help In such a manner that the commitment of the respondent is taken for 
granted. . 



3. The first follow-up letter is brief in reminding the client of his 
lack of response and the importance of his contribution to the evaluation* A 
reminder that he received a stamped, self-addressed envelope for his convenience 
In responding is mentioned. 

4. The essential feature of the second follow-up is the Inclusion of another 
copy of the questionnaire and another stamped self-addressed envelope. The 
accompanying letter notes strongly the mailing of the second questionnaire and 
envelope and is worded in such a way that it is seen as a convenience in the 
event the original one was misplaced. 

5. The third follow-up again contains a reference to the stamped self- 
addressed envelope and invites the client to get in touch with the agency for 
another questionnaire if he has misplaced his copies. 

6. A final follow-up is in the form of a phone call to the respondent In 
which the importance of his contribution to the evaluation process is reemphasized 
and the offer to send another copy of the questionnaire is extended. 

The Intervals between initial mailing and all succeeding follow-ups is an 
extremely Important consideration in the standardization of the data collection. 
Robin (1965) reviewed the research relevant to this and recommended a seven day 
Interval for maximum response. For agencies participating in the Colorado State 
University data bank with widely dispersed geographic area represented by the 
clientele, this recommendation should be viewed as a minimum time lapse with 




eleven days the maximum. 
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Inasmuch as most counseling agencies do at least some routine data collection 
on a monthly basis* the evaluation procedure described here has been designed to 
take advantage of that. By having agency staff members review their case loads 
on a monthly basis and close out those clients no longer being seen* a list of 
terminated clients may be sent questionnaires for evaluation feedback on a monthly 
basis. This not only provides for a control on elapsed time between termination 
of counseling and feedback assessment* but also results in periodic data that may 
be analyzed and/or Interpreted on a month to month basis. 

In order to insure an optimum level of standardization of data collection 
procedures for the Colorado State University data bank* only data collected using 
no fewer than two of the six possible contacts are used. This minimum contact 
may consist of a combination of either steps 1 and 2, a pre-questionnaire letter 
and the questionnaire mailing* or steps 2 and 3, the questionnaire mailing and a 
follow-up. All additional steps that an agency can undertake to Implement are 
encouraged so that as high a response rate as possible will be achieved. 

The critical role of record keeping and data collection in counseling center 
efforts to maintain relevant effectiveness has been explained* and the Importance 
of a data bank in utilizing that Information acknowledged. The Colorado State 
University data bank has been described In pointing out the salient 



of an effort to provide for the essential Ingredients of a comprehensive* efficient* 
and effective system of ongoing data collection and analysis. An agency that 
participates in the data bank not only adds Important data to the pool already 
in existences but also receives a summary print out from every other participating 
institution for use In comparative evaluations. In this way an agency’s efforts 
In the direction of service evaluation may be carried out utilizing both 
Institutional feedback as well as nationwide standardised feedback from other 
institutions. The effective use of both feedback sources can greatly enhance 



agency and Individual competence. 
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Summary 

Counseling center efforts to establish and maintain relevant effectiveness 
in the services they offer must be based on valid and reliable feedback from 
critical sources. Record keeping and data collection are the procedures by which 
this feedback can be acquired if they are designed to answer questions and do not 
become ends in themselves. A follow~up questionnaire and mailing procedure 
designed to provide both descriptive and evaluative data can be most useful in 
producing data useful in a number of Important ways in the assessment of agency 
response to environmental needs. Cross**agency comparisons have great potential 
and emphasize the Importance of the development of a data bank equipped to 
handle both descriptive and evaluative material . The Counseling Services Assess- 
ment Blank was recently developed and adopted, along with a standardized mailing 
for the evaluation of counseling center feedback through a data bank being 
developed at Colorado State University. This data bank will make possible cross- 
agency comparisons of evaluative data for the first time. Feedback assessment 
of the typd described here is essential for Improvement in the effectiveness of 
both individual staff and agency programs. 




For additional information concerning the Colorado State University Data Bank 
write: Rocky Mountain Behavioral Sciences Institute, Inc., P.O. Box 2037, Fort 

Collins , Colorado 80521 
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Figure 1 Problem-Goal and Cause Dimensions of the 
Missouri Diagnostic Classification Plan 
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